A.i. in Real-time (automated and Interactive) Sonification (of Gesture Data) Generative Real-time Sonification A.i. in Real-time (automated and Interactive) Sonification (of Gesture Data)

Drawing on implementations of A.I. models for real-time music performance that produce an immediate sonification and visualisation display from gestural and audio input, this paper concerns three areas of relevance to a broader array of inter-domain implementation: integration of A.I. modules (Artifical Life, biologically-inspired and Evolutionary processes) in a system for real-time computational data processing; using A.I. modules for sonifying data – for automated and interactive generation of sound; and considerations for mapping data to auditory display using spatial and timbral (tone colour) dimensions to increase the efficacy of complex data streaming and recognition. Automated and interactive generative structures in sonification are metaphorically akin to automated drawing of sketches or designs from structural models and using sketching for interactive design synthesis. The significance of this approach is the applicability of its data mapping principles to a range of sonification contexts, using generative processes to synthesise and perpetuate sound in real-time, and interchangeable generative modules in the system mediating between designer/user/interaction and representation. The example presented uses Evolutionary Looming to scale frequency as a consequence of input loudness and noisiness, a Neural Oscillator Network to perpetuate sounds with concordant pitch (frequency) derived from auditory input, and gestural interaction to adjust parameters of granular synthesis and the generative processes. Auditory display is an emerging modality for data representation, both for use alone in visually heavy contexts, where sonification presents an effective alternative to visualisation, and in bi-modal audio-visual display environments where sonification can reinforce other modalities, enhancing fidelity of representation and reasoning based on it. Due to the interplay of auditory cognition, memory and inherently time-based representation of sound, sonification can provide superior recognition to visualisation for certain types of features, such as periodicity, discrete irregularities, subtle shifts over time, stream segregation and very fine increments of data represented using frequency.